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Abstract: Education refonn efforts have mandated the use of student achievement data in 
schools. This Q-methodology study investigates the perceptions of principals and teachers 
about how data are used or misused. Principals in the sample were found to use data mostly 
to evaluate the school, make improvements, and model best practices of data use. Teachers 
used data to improve instruction and outcomes for students. Results indicate a need to 
create an assessment savvy environment where data are used to improve practices. 
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1. Introduction 

The 2001 reauthorization of the Elementary and Secondary Education Act (No Child Left Behind, 
NCLB) formalized a movement to attach leader, teacher, and school success with measures of student 
achievement. The catalyst of this movement can be attributed to the Cold War with the Soviet Union in 
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the 1970s and 1980s and the economic race to keep pace with Asia (specifically Japan) in the 1990s. 
Today global competitors (e.g., China, India, etc.) have promised even more efforts to keep American 
students prepared for a flattened world [1]. As a result, today’s metric to assess school effectiveness is 
student achievement; merging assessments, data, and accountability. However, data can have the 
potential to be valuable in other ways. To begin, data can provide valuable feedback loops to teachers 
to refine their pedagogical practices and thus improve student learning. Additionally, data can be used 
to uncover inequities. The NCLB legislation has done both. Interestingly, NCLB has indeed proved 
valuable to uncover (i vis-a-vis disaggregated achievement data) inequalities in learning among sub¬ 
groups. How NCLB has impacted school level educators’ leadership and pedagogy is a matter of 
debate and a focus of this study. 

Increased accountability is one way the government is coping with increased global competition 
and other pressures. As such, accountability is manifested in the form of high stakes testing, common 
core standards, and the surge in the number of charter schools and other educational alternatives. 
Federal and state governing agencies, along with the private corporate sectors, joined to voice calls for 
a standard core curriculum and assessments that measure progress of such a curricula. The curricular 
work was realized only in part. That is, as a result of the 1980s Secretary of Education’s 1982 report, A 
Nation At Risk, the demands placed on students’ curricular requirements rose dramatically in short 
order (e.g., high school students around the country were required to take math, science, and foreign 
languages classes). It would take another three decades before the common curricula would be realized 
(in 2010 the common core standards for mathematics and English were approved by 45 states (Haager 
and Vaughn, 2012); and common standards for social studies are currently underway). Prior to the 
common core there was no set measure to student achievement on a national comparative basis. 
Common core standards provide consistent assessment measures with which to compare student 
achievement nationally. Scores from common core assessments will then be used as data toward 
assessing the quality of teaching and learning occurring in schools. 

This educational reform effort was manifested into the acceleration of assessment systems in the 
1990s. In this decade state Governors and national business leaders convened for separate initiatives to 
press for the improvement of education in the name of economic competition. Both initiatives called 
for more data to be used to monitor school effectiveness in general, and more specifically, individual 
student achievement. In 2001, No Child Left Behind (NCLB) was a historic school reform measure 
enacted from the federal level to create a new normative assessment practice in K-12 schools [2]. 
NCLB created specific achievement targets and mandated specific measures to monitor the growth 
toward the targets. Moreover, sanctions were developed as enforcements for educators failing to meet 
the student achievement targets. As a result, schools have been introduced to the world of 
assessments—formative and summative, diagnostic, data warehouses, and data-driven 
decision-making—to name a few. 

In spite of these new accountability measures, there is a widespread belief that the educational 
profession has underutilized the valuable commodity of student achievement data to inform how and 
how well students are learning. The mandates to use data left out the declarative (why), practical 
(how), and procedural (why), knowledge needed by district and school level educators. In 2002, the 
U.S. Department of Education unabashedly expressed its goal to “change education to make it an 
evidence-based field” building upon the belief that education has been operating “largely on the basis 
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of ideology and professional consensus” and, “as such ... is subject to fads and is incapable of 
cumulative progress that follows from the application of the scientific method” [3]. Popham stated that 
“given the significance of students’ test scores these days, teachers and administrators who choose to 
remain unaware of assessment’s key concepts are being dangerously naive” [4], Unfortunately, 
educators find purposefully using school data a difficult skill to master- despite increases in high 
stakes and ever increasing rates and measures of accountability [5-13]. 

Nevertheless there continue to be a steady stream of data delivered to the schoolhouse door and a 
constant pressure to “use data.” As a result, schools have been characterized as placed in “the 
paradoxical situation of being both data rich and information poor” [14]. What do practices look like 
for current school educators? How do they perceive their uses of school data as mandated by federal 
law (NCLB) and district and school policy? How do school level leaders perceive the use of data? The 
aim of this study is to gain an understanding of how teachers and principals are using data. We chose 
teachers and principals as the units of analyses because they are in the crosshairs of both the mandates 
to use data and the associated sanctions. The use of data has only escalated since NCLB. For instance, 
there is a clear and present movement to use student achievement data to evaluate, reward, and punish 
teachers. Understanding how these school level educators have come to understand and use data is 
both timely and important. 

Although the terms data and assessment have taken on a negative connotation due to the increased 
pressure created by high stakes testing, assessments can also be used to directly, immediately, and 
positively impact student learning. Yeh defines rapid formative assessment as, “systems that provide 
formative feedback to students and teachers regarding student performance. Assessments are dubbed 
‘rapid’ because they are provided two to five times weekly. When conducted to promote student 
growth and learning, assessment is aligned with strategically selected key intended outcomes, and 
integrated into daily instructional activities” [15]. 

2. What Are Data? 

The challenge for building data use capacity and clarifying the ambiguity during associated 
activities stems from a lack of literacy with respect to the use of data in practice. There is confusion 
when referring to data; as different definitions, fonns and contexts for its use exist. Examples of broad 
categories of data that educators access in education daily are summative and formative. Summative 
assessment data are typically derived from end of the year assessments that monitor and record student 
achievement [16,17]; and they provide a macro-level vantage point most useful for educators and 
policy makers at a distance from the classroom [18]. These data are either from the criteria-based 
assessment (e.g., NCLB approved state-assessment), nonn-referenced assessment (e.g., TIMS—Trends 
in International Mathematics and Science, PISA—Program for International Student Assessment, or 
NAEP—National Assessment of Educational Progress). Fonnative assessment data are then used for 
“shaping student’s learning.” [19] These activities describe “a planned process in which assessment- 
elicited evidence of students' status is used by teachers to adjust their ongoing instructional procedures 
or by students to adjust their current learning tactics” [20]. In other words, summative data are autopsy 
data that are collected after the fact while formative data are biopsydata that are still actionable. 
However, no two formative assessment systems are exactly alike. For example, Confrey and Kazak 
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[21] contend that, while it was once thought that a singular learning trajectory existed for all students, 
variants of student trajectories exist that must be reconciled as teachers teach and students leam. 
Certain math assessments provide teachers vital insight into these individual learning trajectories. In 
the end, mathematical learning trajectories may “increase the likelihood of successful learning by 
defining a corridor of possible opportunities and constraints and recognizable typical patterns for use 
by teachers in conducting practice” [21]. As a result, logical progression can be mapped to model 
student cognition. This gets at what students know (the content), and how they know it, (the route they 
took to achieve competence). 

Young and Kim stated that the, “purposes that assessment data are supposed to serve and the 
occasions for using the data are not necessarily defined a priori ; teachers choose how and when to use 
the infonnation gathered through assessments” [22]. The knowledge management field has 
investigated the transformation of raw data into meaningful information in order to make 
knowledgeable decisions about one’s practice [7]. 

Militello and colleagues have conducted a series of in-depth case studies in order to analyze what 
data are used in schools, how they are used, and if they are valid measures for each associated use. 
Results indicate that (1) many assessment systems broadly define the validity of their products, 
promising more than their psychometric properties can deliver, and (2) the use of many assessment 
systems are misguided, as the end-users are uninformed of the scientific characteristics of each 
assessment. To help school educators, Militello and colleagues created an Assessment Fit Principle 
[23]. The principle states that there must be a match between the purpose of the assessments (e.g., 
properties of assessment including validity) and the intended uses by school educators (e.g., lesson 
planning) [13,23,24]. Effective use of assessments for educational decision-making is predicated upon 
this fit. Figure 1 provides a visual representation of the concept of fit. The line of fit in the figure li nk s 
specific types of data with particular end-users. This figure was developed from a series of empirical 
studies [13,23]. These studies documented the use of various assessment systems (e.g., National 
NEAP; state test; third party formative; district benchmark) and analyzed this use within the 
framework of the product’s validity. As a result, a qualitative “line of best fit” was hypothesized for 
these districts. 
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Figure 1. Assessment Fit Principle [23]. 
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What emerged from this work were cognitive diagnostic assessments that provide teachers detailed 
information regarding how, and if students are learning. Leighton and Gierl define cognitive diagnostic 
assessment (CDA) as a means to “measure specific knowledge structures and processing skills in 
students so as to provide information about their cognitive strengths and weaknesses” [25]. “Fit for the 
classroom-level” resides at the diagnostic level in the assessment system axis. Diagnostic 
characteristics track student learning and knowledge at a fine grain level, and at a faster rate than 
before [18]. By giving teachers cognitively diagnostic data in a timely fashion, teachers can change 
their teaching in response to the data. This allows data to be transformed from assessments of learning 
to assessments for learning [26-28]. That is, assessing for learning stipulates that assessment data fits 
classroom educators’ need for diagnostic-level and timely information about their students. Armed 
with relevant and timely data, educators who were formerly inept at efficiently operationalizing the 
results of student data, are now able to use data in their practice. Studies have supported the use of 
assessment for learning to impact student learning [29-31]. 

3. Promising Use of Data 

There is a growing body of literature on the promise of data use. For example, the U.S. Department 
of Education stated, “Research shows that teachers who use student test performance to guide and 
improve teaching are more effective than teachers who do not use such information” [32]. Black and 
William extensively reviewed 250 studies to ascertain the influence of formative assessment on 
learning [33]. They describe impressive learning gains within the classroom, especially when 
formative feedback is at the teacher-student interaction level. Students taught by teachers who use 
assessment for learning achieved in six to seven months what otherwise took one year [28,33]. Shute’s 
meta-analysis of formative feedback revealed .40 to .80 SD gains in student learning. “Effective and 
useful feedback,” she argued, “depends on three things: (a) motive (the student needs it); 





Educ. Sci. 2013 , 3 


103 


(b) opportunity (the student receives it in time to use it); and (c) means (the student is able and willing 
to use it)” [34]. Finally, cost-effectiveness studies have always been the gold standard of research, 
second only to student outcome. Yeh found that rapid formative assessments are 124 times more 
cost-effective than class size reduction [15]. 

Studies found that effective use of data in schools included multiple sets of data as well as 
collaboration, especially between teachers and principals [35], Leadership has been a key ingredient to 
innovation and reform in general, but there is evidence that leadership has a pronounced effect on 
teacher’s use of data [14,36-41]. Massell also found that increased communication and knowledge 
provided by data appeared to be positively altering educator attitudes toward the school’s capabilities 
of educating some underperfonning groups [42], These evidences demonstrate some of the positive 
effects of data use on teachers’ instructional practice and school improvement. The extant educational 
literature has generated nonnative features of the meaningful, effective use of school data. Specific 
outcomes of using data have included: 

• Increases reliability and predictive validity of school improvement efforts [5,13,14]; 

• Helps improve instructional practice [5,14,43-45]; 

• Monitors teacher effectiveness [24,46]; 

• Targets student needs for short-tenn intervention [15,24,37,43,47]. 

School leaders drive the success of their schools as they integrate appropriate data usage into 
shaping their vision. When schools have a clear vision, this encourages a data culture where students 
themselves can be involved in taking ownership of their own data [48], Simultaneously, there are 
renewed warnings that data has limitations for teacher evaluations [49], and that careful consideration 
must be given to the fit between the intended uses and psychometric purpose of data [23]. 

4. Interrogating Assessment Knowledge: Implications for Pedagogy 

In the end, the relevancy of data for educators is rooted in the integration into instructional practice, 
accessibility of informative data, ability to make sense of data into meaningful information, and 
especially—their own general and pedagogical content knowledge. Additionally, organizational 
conditions and setting/contexts matter [22]. Much the same way as technology pedagogical content 
knowledge (TPCK) was established [50]—an assessment pedagogical content knowledge (APCK) 
is needed. 

Coburn and Talbert (2006) note that psychometric properties of the assessment, alignment with 
desired outcome, ability for assessment data to give insights into student thinking and reasoning, 
alignment of results with their own knowledge, and the use of multiple sets of data is what makes 
evidence valid for individuals. This summary aligns with the ‘Fit Principle’. When grading student 
work, teachers use more than just the fonnal measures [51]. Most teachers also utilize internally 
created assessments to supplement the results received from more formal, mandated measures [13,52]. 

Teachers test frequently, using tests they created for their assessment purposes, and feel put off by 
additional tests. They are also particularly unsatisfied when forced to administer published tests that 
they did not help create. Teachers oppose such assessments because they most often are not aligned 
with teachers’ content and style, and are thus incapable of accurately measuring student growth and 
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learning. Their frustration is justified, as assessments are a waste of valuable time when they do not 
produce valid and reliable data [22,53]. 

Challenges to changing school cultures must be addressed if student assessments are to have a vital 
role in classroom teaching. While existing studies on teacher and school level data use focus on the 
indications of successful implementation, an understanding of school or classroom level data use 
remains more opaque. This investigation sheds light on teachers and principals’ perceptions of their 
use of data. Q-methodology was used as a tool to help us understand (1) why teachers and principals 
are using the data in the ways they are and (2) how the use of data are or are not supported. The 
development of q-methodology as well as the collection and analysis of q-sort data will be discussed in 
the next sections. 

5. Research Design 

Q-methodology is a research methodology that allows the researcher to study the perceptions of 
individuals [54]. Specifically, the methodology allows for a process of analyzing multiple perspectives, 
where each perspective is highly idiosyncratic and valued [56]. Q methodology invites participants to 
make decisions as to “what is meaningful” and hence, what does (and what does not) have value and 
significance from their perspectives around a given subject. When using Q methodology, the 
researcher asks participants to sort a collection of statements about a topic from the most important or 
characteristic statement to the least important or characteristic statement for the topic that is being 
investigated (e.g. are teachers using data to modify their lesson plans). The statements that are sorted 
are referred to as a Q-sample and are often generated through interviews with people who are 
knowledgeable about the topic at hand [57]. Q methodology was created as an attempt to understand 
subjective thoughts in an objective manner. The purpose of Q methodology is to better understand the 
relationships of subjective statements by a group of participants. While data obtained from the use of 
Q-methodology is not generalizable, it is both transferable and informative to that particular group 
of participants. 

5. 1. Q Sample Statement Development 

The Q sample statements are a principle component of Q methodology, and represent “a collection 
of stimulus items . . . that is presented to respondents for rank ordering in a Q-sort” [57]. For this 
study, a concourse of statements was generated from the extant literature on the use of school data and 
interviews with principals and teachers. A pensive review of related literature and interviews led to the 
development of a set of statements that best reflected the promising use of school data by principals 
and teachers. Initially, more than 50 statements were developed for both teachers and principals. A 
small set of principals (5), and teachers (6), were then interviewed about their uses of school data. This 
helped us add to the body of statements derived from the literature. Finally, the statements were 
analyzed for redundancy and uniqueness, and ultimately edited for clarity. In the end, this concourse 
resulted in 23 card statements that represented the Q-sample for both the principal and the teacher 
study (see Table 1 and Table 2). 
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Table 1. Principal Q-sample. 


1. I have easy access to school data. 

2. I have developed a common language and culture about school data with my staff. 

3. I ensure that teachers receive or can generate useful school data reports. 

4. I use school data to allocate resources. 

5. I ensure that teachers are provided ongoing support and professional development for the use of school data. 

6. I model how teachers can use school data to inform their practice. 

7. I use school data to evaluate teachers. 

8. I collect, analyze, and use school data in our school improvement plans. 

9. I collect, analyze and use parent, teacher, and student survey data. 

10. I analyze school data using student demographic variables. 

11. I recognize and celebrate accomplishments based on school data. 

12. I use school data to create new programs for students. 

13. I designate a data specialist who assists teachers in the use of data to inform instruction. 

14. I collect, analyze and use school data to evaluate school programs. 

15. I ensure that assessments (other than the state assessment) have been aligned with state standards. 

16. I hold teachers accountable for using school data to inform their teaching practice. 

17.1 ensure that teachers receive assessment data results within one week of administration, (not including MCAS, 
SAT) 

18. I ensure that we use a centralized technological system to store, access and analyze data. 

19. I structure regular, designated time for teachers to collaborate with colleagues about school data. 

20. I am evaluated based on school data. 

21. I ensure that teachers have easy access to school data. 

22. I ensure that parents have access to data. 

23. I support teachers in their use of data. 


Table 2. Teacher Q-sample. 


1. I regularly use school data to develop and modify lesson plans. 

2. I regularly use school data to recommend students in remedial programs. 

3. I regularly use school data to set performance goals for students. 

4. I regularly use school data to communicate with colleagues about student learning. 

5. I regularly use school data to form flexible student groups in the classroom. 

6. I regularly use school data as evidence at meetings (e.g. pre-referral, 504, IEP). 

7. 1 regularly use school data to modify my instructional strategies (e.g. differentiate my instruction). 

8. I regularly use school data to recommend students’ placement in advanced or gifted and talented programs. 

9. I regularly use school data to identify content to re-teach. 

10. I regularly use school data to communicate with the student’s home. 

11. 1 regularly use school data to communicate with my grade-level/content area colleagues about teaching practices. 

12. I regularly use school data to form flexible student groups with other teachers during the school day. 
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13. 1 regularly use school data to communicate with colleagues outside my grade-level/content area. 

14. I regularly use school data to focus on students that are not yet, but close to, passing the state assessment. 

15. 1 regularly use school data to communicate with school and/or district administrators about teaching practices. 

16.1 regularly use school data to communicate with students about their progress towards learning standards. 

17.1 regularly use school data to identify my own professional development needs. 

18.1 regularly use school data to monitor student progress toward curricular standards. 

19. I regularly use school data to monitor the progress of students by sub-group in my classroom (e.g. student 
demographics). 

20. I regularly use school data to compare my students with my colleagues’ students. 

21. I regularly use school data to assign student grades. 

22. I regularly use school data to advocate for resources for my students. 

23. I regularly use school data to communicate with school and/or district administrators about student learning. 


6. Data Collection 

6.1. P Sample and Data Collection 

Participants were asked to rank order the 23 card statements as well as to complete a questionnaire 
after they performed the sort. Thirty-four teachers completed the teacher q-sort and the subsequent 
open response questions, while twenty-eight principals completed the principal q-sort and subsequent 
open response questions (see Appendix for the protocol). 

The conditions of instruction for sorting the Q-sample asked the participants to rank order the 23 
statements (printed on 3x1 cards) on a continuum of “most characteristic of how I use school data” to 
“least characteristic of how I use school data.” Responses were recorded on a grid containing nine 
columns with headings ranging from +3 (most characteristic statements) on the right side to -3 (least 
characteristic statements) on the left side. Each grid had two spaces available under the end points, 
three under +1-2, four under +/-1, and five under 0. In parlance with Q methodology, the distribution 
was created to resemble a normal curve (see protocol in Appendix to see the sorting grid). In addition 
to perfonning the Q sort, participants were asked a series of questions regarding decisions they made 
in performing the Q-sorts related to why they sorted the Q-sample as they did (e. g., “If there were 
other specific statements that you had difficulty placing, please list the number of the statement and 
describe your dilemma”). 

7. Data Analysis 

7.1. Principal Sort Data Analysis 

Initially, the principal study produced three emergent factors. The eigenvalues and corresponding 
explained variance demonstrated no clear factors. However, analysis of the Scree plot displayed a 
strong initial factor. For this study, two factors were rotated using the Varimax rotation technique. 
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Factor A accounted for 16% of the explained variance. Seven of the 28 principals sampled (25%) 
expressed a significant degree of similarity within the statements provided in the sort. In 
Q-methodology, significance is identified using the following formula: 1/square root (N) X 2.58 
(standard deviation, p<.01) where N is the number of cards. Consequently, the significance value for 
the principal study was 0.54 (see Table 3). 

Table 3. Principal factor loadings matrix. 


Factor Loadings 


Principal 

Factor A 

Principal 

Factor A 

Principal 

Factor A 

01 

-.20 

Oil 

-.03 

M7 

.09 

02 

.41 

012 

.69* 

M8 

.67* 

03 

.54* 

013 

.37 

M9 

.31 

04 

.16 

014 

.38 

K1 

.38 

05 

-.02 

015 

.11 

K2 

.30 

06 

.49 

M2 

.26 

K3 

.32 

07 

.55* 

M3 

.45 

K4 

-.30 

08 

.36 

M4 

.63* 

K5 

.07 

09 

.65* 

M5 

.26 



010 

0 

M6 

.54* 




Note: * indicating a defining sort (p<.01) 

7.2. Teacher Sort Data Analysis 

The teacher study produced eight emergent factors. As with the principal sort analysis, the 
eigenvalues and corresponding explained variance demonstrated no clear factors. Analysis of the Scree 
plot, again, displayed a strong initial factor. For this study, two factors were rotated using the Varimax 
rotation technique. Factor A accounted for 41% of the explained variance. Thirteen of the 34 teachers 
sampled (38%) expressed a significant degree of similarity within the statements provided in the sort. 
The significance value for the teacher study was 0.54 (see Table 4). 

Table 4. Teacher factor loadings matrix with an * indicating a defining sort (p < .01). 


Factor Loadings 


Teacher 

Factor A 

Teacher 

Factor A 

Teacher 

Factor A 

SI 

.55* 

Ml 

.02 

AA2 

.22 

S2 

.51 

M2 

.19 

Cl 

.40 

El 

.32 

M3 

.25 

C2 

.42 

E2 

.62* 

J1 

.17 

PI 

.47 

N1 

.51 

AB1 

.57* 

P3 

.68* 
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A1 

.67* 

AB2 

.38 

D1 

.09 

A2 

-.04 

AMI 

.64* 

D2 

.01 

A3 

.59* 

AM2 

.24 

J2 

.17 

A4 

.59* 

R1 

.51 

AB3 

.08 

K1 

.73* 

R2 

.15 

MG1 

.69* 

K2 

.07 

AA1 

.57* 

MG2 

.75* 





P4 

.61* 


Note: * indicating a defining sort (p<.01) 

For purposes of this research, we decided to report on the main or most significant factor for 
principals and teachers in this study. As a result, we describe and compare the model factor A for both 
principals and teachers. 


8. Findings 

The exploratory analysis of the principal data produced a main factor. Seven study participants 
returned statistically significant strong positive loadings on one factor. This indicates that these 
principals were in agreement on several characteristics of how they use data. The statements listed in 
the table below with a q-sort value of +3 are those that these principals found to be most characteristic 
of their practice. The statements with a q-sort value of -3 are the ones principals thought were least 
characteristic of their behavior with respect to data use. 

Table 5. Most and least characteristic statements for principal factor. 


Q-Sort Value 

Statement 

+ 3 

As a principal, I have easy access to school data, (card #1) 

+ 3 

As a principal, I collect, analyze, and use school data in our school 
improvement plans, (card #8) 

+ 2 

As a principal, I collect, analyze and use school data to evaluate school 
programs, (card #14) 

+2 

As a principal, I model how teachers can use school data to inform their 
practice, (card #6) 

-3 

As a principal, I ensure that assessments (other than the state 
assessment) have been aligned with state standards, (card #15) 

-3 

As a principal, I designate a data specialist who assists teachers in the 
use of data to inform instruction, (card #13) 


The statements placed by the principals at the positive end of the spectrum show that they use data 
to analyze and evaluate their schools and make improvements as well as for modeling best practices of 
data use. 

Upon observing the protocols for the principals loading statistically significant regarding the 
statements that they agreed upon one principal said, “This semester’s plans have centered around the 
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student survey results from two different surveys” (MM8). Another principal said, “I use the data I 
have to develop class reports for my teachers to show them how their students have performed in the 
past and what their chance is of doing well in their class. I also use it to ID (identity) students that are 
at-risk and show teachers how to do the same” (MM6). In reference to the importance of data in 
school, one principal stated, “My years of experiences as an administrator, along with my school’s 
involvement in multi grants, gives me the perspective that data are very important in schools. 
Although, initially teachers were overwhelmed and somewhat disinterested in the data, they are now 
quite adept at reviewing/discussing and utilizing the data effectively” (OD7). These statements support 
the conclusion drawn by the main factor, which indicates principals use data to make school 
improvements and to model best practices of data use for their teachers. 

Alternatively, principals placed card 13 that says, “As a principal, I designate a data specialist who 
assists teachers in the use of data to infonn instruction,” at the negative end of the continuum. One 
principal said he placed it at the end because, “This is not easily accomplished because there is just one 
data specialist in our district whose availability seems limited to helping our school and district 
administrators interpret reports. Then it becomes my responsibility, as a principal, to consult with 
elementary curriculum specialists who can help me further interpret the data and provide support to 
staff. This is most definitely a budget issue, and since one person can’t do it all, I think it is important 
that principals become more knowledgeable and skillful with regard to how they can provide support 
to teachers in the use of data to inform instruction.” Another principal said he placed card 15 that says, 
“As a principal, I ensure that assessments (other than the state assessment) have been aligned with state 
standards” at the negative end of the continuum because “I needed to separate my actual practice from 
what I know and believe to be best practice.” This principal’s statement basically means he struggles 
with aligning assessments with state standards, in other words teaching to the test. We can conclude 
from the analysis of cards at the end of the continuum as well as the statements above that principals 
do not have the support of a data specialist to assist them and their teachers in adhering to demands to 
use data being exerted upon them by policy created above the school level. 

The exploratory analysis of teacher data produced a main factor. Thirteen participants of the teacher 
study returned statistically significant strong positive loadings on one factor. This indicates that these 
teachers were in agreement on several characteristics of how they use data. The statements with a q- 
sort value of +3 are those that these teachers found to be most characteristic of their practice. The 
statements with a q-sort value of -3 are the ones teachers thought were least characteristic of their 
behavior with respect to data use. 
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Table 6. Most and least characteristic statements for teacher factor. 


Q-Sort Value 

Statement 

+ 3 

I regularly use school data to develop and modify lesson plans, (card #1) 

+ 3 

I regularly use school data to modify my instructional strategies (e.g. 
differentiate my instruction), (card #7) 

+ 2 

I regularly use school data as evidence at meetings (e.g. pre-referral, 
504, IEP.) (card #6) 

-3 

I regularly use school data to compare my students with my colleagues’ 
students.(card #20) 

-3 

I regularly use school data to form flexible student groups with other 
teachers during the school day. (card #12) 


The statements placed by the teachers at the positive end of spectrum show that teachers use data 
mainly focusing on improving their individual instructional practice by modifying lesson plans, 
instructional strategies and re-teaching content. Moreover, school data is also used by teachers in a 
formative manner to improve the outcome of students, such as giving the feedback to students and as 
the evidence to make the decision at the meeting. 

The relation of these statements to each other clearly centers upon data’s utility to inform classroom 
practice. A number of the corresponding open-ended response questions illustrate this tendency as 
well. One teacher commented: “This is the main purpose of my assessments. I try to use them to focus 
my instruction on weak/difficult areas or topics that need to be re-taught” (SMI). Another teacher who 
loaded significantly commented that, “Sometimes data helps us drive instruction and to teach and get 
better results” (RC1). In addition respondent said, “The more fonnal assessments we have to support 
the need for assistance the stronger a case we have. We need to be able to prove that a child is placed 
where they are or are receiving services for a reason. This testing also helps teachers detennine when 
services are not appropriate or needed” (NF2). These statements in conjunction with the cards placed 
on the positive end of the continuum support our conclusion that teachers are using data to improve 
instructional practice. 

In contrast teachers indicated that they did not regularly use data in making decisions on how to 
interact with other teachers. For example, teachers said they placed card 12 that states, “I regularly use 
school data to form flexible student groups with other teachers during the school day” on the least 
characteristic end of the continuum because, “I don’t have any time to see other teachers in the school 
day” (C02). Another teacher said she placed card 12 at the negative end because, “We never do this. It 
sounds really cool and I’d like to do it. I think the pain of not doing it is what landed it in the -3 
column” (P3). 

One teacher said she placed card 20 that says, “I regularly use school data to compare my students 
with my colleagues’ students” at the end of the continuum because, “I rarely, if ever, compare my 
students to other colleagues / students. Personally, I don’t like to compare students but to look at them 
as individuals, and their personal success/challenges” (P4). These statements from teachers are 
indicative of the ambiguous environment within which teachers are making decisions about how to use 
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data. Though the literature illustrates that effective use of data includes using it to initiate 
conversations and to support collaboration among teachers, teachers do not feel they have the time and 
some do not see the need to use data in this way. 

The emerging factors do describe to some extent why and how teachers and principals are using 
data. The answer to the question of how the use of data is or is not supported is alluded to in the open- 
ended statements made by principals and teachers. These statements tended to lean toward comments 
about not having the necessary time and technical support to use data in some of the ways the literature 
states as effective. As seen above, principals and teachers did address how the use of data are or are not 
supported in their answers to the open-ended questions. 

9. Discussion and Conclusion 

Data are becoming increasingly important in the data driven climate educational workers are 
currently required to function in. Educational leaders are grappling with national and state summative 
assessments, district benchmark assessments, and the emergence of a multitude of fonnative 
assessment products. Making these data an integral, systemic part of their professional practice will 
hinge on acquiring the knowledge, skills, and dispositions to understand assessments as the means, not 
the ends of the instructional process. That is, school leadership and assessments and instruction must 
be integrally linked, not diametrically opposed [58-60], Studies have demonstrated that teachers can 
find data meaningful when they are able to use the data to inform what they know about students, and 
consequently what they do [61]. Additionally, while educational reform and change take an emotional 
toll on educators, when they are able to leam, see, and experience the success with a reform, they will 
form emotional and intellectual bonds with the new approach. 

The knowledge management field has investigated the transformation of raw data into meaningful 
information in order to make knowledgeable decisions about one’s practice [7]. The resources needed 
for such a transformation include “people, processes and technologies” [8]. Bringing these elements 
together means the creation and support of an assessment-savvy learning organization. Educators in 
such an environment can transform accountability systems into real changes in professional practice 
“to the extent that they generate and focus attention on information relevant to teaching and learning, 
motivate [people] to use that information and expend effort to improve practice, build the knowledge 
base necessary for interpreting and applying the new information to improve practice, and allocate 
resources for all of the above” [62]. 

Federal and state level policy demands for data use create implications for principals and teachers. 
The commonalities discovered through this study indicate that both principals and teachers agreed that 
they used data to improve and inform their individual practice. However, the differences exhibited by 
the card statements placed at the end of the continuum coupled with the open-ended statements paint a 
picture that demonstrates a lack of support and time to meet the demands of policies that are left up to 
the interpretation of the practitioner. 

Results demonstrate that the most important aspect of data use for teachers is making classroom 
decisions. However, policies such as NCLB and the implications of AYP mean that teachers 
expanding their focus to the implications of data use for the entire school would prove to be more 
powerful. An integrated approach to developing strategies for implementing data use in schools could 



Educ. Sci. 2013 , 3 


112 


enable teachers and principals to collaborate and understand the best use of data for their particular 
school setting and create the coherence to replace the disconnectedness. While the results of the 
principal study demonstrate that principals focus on using data to evaluate programs and school 
improvement, principals could positively impact their teacher relationships by expanding their focus to 
providing opportunities for teachers to leam more about their expectations for using data through 
professional development that involve both parties in setting goals for data use. 

Inside the paradox created by the ambiguity of teacher and principal data use and the unrealized 
connection are embedded problems such as defining data and data use, implementation of effective 
data use, building teacher and principal data use capacity, and overall institutional accountability. That 
is to say, merely adding assessment data to schools will not necessarily lead to student achievement 
gains and overall school improvement. 

The findings of this study indicate that we cannot assume we are talking about the same data unless 
we have purposely implemented a systemic approach to data use that proves to be relevant to both 
teachers and principals. Additionally, motivation, and knowledge, resources and support {capacity), 
and the valid and meaningful institutional measures and standards ( coherence ) are necessary to 
transfonn practice [58,62]. 

While the results of this study did answer the question of how teachers and principals are using 
data, there are more questions that remain. For example, in this study the definition of data given to the 
respondent was broad and left to the interpretation of the respondent. In future studies asking teachers 
and principals to define the types of data they are using to make decisions for their practice might 
produce more specific results and be more helpful to practitioners. In addition, further studies should 
be conducted where the review of literature and interview questions asked in development of the cards 
have in mind the purpose of comparing and contrasting how teachers and principals use data. Lastly, 
an analysis of the emergent factors with respect to individual and school demographics might also 
provide an interesting perspective of teacher and principal data use. 

Generally, when introducing a new curriculum or policy change rarely can we expect the 
stakeholders to follow our lead without a foundational plan. Knowledge of types of data and how to 
effectively use the data to achieve stated outcomes should be part of the process for educating 
principals and teachers. In addition teachers and principals can work together to eliminate the 
confusing view of data by purposefully collaborating on the development of goals and objectives for 
using data in their individual schools. Since the goal of q-methodology is to objectively analyze 
subjective thoughts of individuals the results of this study is both transferable and informative to the 
group of study participants. Militello and Benham found that q-methodology is a useful tool for 
mobilizing discussions, engaging in reflection, and planning new activities [63]. Therefore, the 
application of q-methodology to the topic of data in a particular school setting could be the key to 
helping principals and teachers better understand and navigate the data environment in their school. 

The levels of accountability and responsibility between teachers and building level principals are 
different; consequently teachers and principals conceptually use school data differently. As such, 
principals have the greatest impact on teacher perfonnance; while teachers have the greatest impact on 
student learning and achievement. In order to be effective, principals need to understand and use data 
in ways that help them see a snapshot of both student learning and teacher effectiveness, and to use this 
image to inform their leadership. Teachers need to understand how to use school data in order to tailor 
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their instruction toward increased student learning. In short, both teachers and principals can benefit 
from a more in-depth knowledge and understanding of data usage in order that they might select the 
right data and learn from it. We conclude that in order to improve the quality of data use among and 
between teachers and principals, school leaders must focus on creating a comprehensive plan for data 
use in their schools, which includes creating a data literate environment. This data literate environment 
starts by investing in development of the schools human capital by providing time for teachers and 
principals to collaborate as well as by providing the technical support which provide practitioners with 
knowledge and in turn bring clarity to the how and why of data use. 

This study demonstrates the importance of the investment in training principals and teachers in 
understanding and in the proper usage of school data. Without such data literacy, the desired amplified 
and cohesive use of data cannot occur. That is, the use of data toward more efficient and effective 
school operations that leads to the ultimate goal of increased student achievement will not be possible. 
It is clearer than ever before that only those literate and proficient in data usage will be able to survive 
in this era of accountability in which we currently operate. If we are to expect advances in pedagogical 
practices toward equitable student achievement ends, then how school leaders and teachers understand 
and use school data are of paramount importance. The road to improve teacher pedagogy and equitable 
student learning may indeed be paved with data. 
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Appendix 

School Data Q Sort 

CODE (for interviewer):_ 

1) Gender:_M [1] _F [2] 

2) Ethnicity (Check One): 

_African American [1] 

_ Caucasian [2] 

_ Hispanic/Latino (a) [3] 

_ Middle Eastern [4] 

_ Native American [5] 

_Asian [6] 

_ Other:_[7] 

3) Current Position: 

_ Principal [1] 

_ Assistant Principal [2] 

_ Teacher [3] 

_ Other (Please Specify :_) [4] 

4) Current Work Setting: 

_ Elementary School [1] 
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_ Middle School or Junior High School [2] 

_High School [3] 

_Combined Middle and High Schools [4] 

_ Other (Please Specify :_) [5] 

5) Do you consider your school 

_Urban [1] 

_Suburban [2] 

_Rural [3] 

6) Indicate the range which best describes the student population at your school. 

_0-499 [1] 

_500-999 [2] 

_ 1000-1499 [3] 

_ 1500 or more [4] 

7) Current educational level: 

_Bachelor [1] 

_Masters [2] 

_Masters +30 (or Certificate of Advanced Graduate Study) [3] 

_Doctorate [4] 

8) How many years have you worked as an educator?_ 

Q Sort Instructions: 

1. Throughout the sort keep the following definition of School Data in mind: 

“School data are defined as any formal information about students received or generated. For 
example: MCAS reports, survey results, psychological testing, district-sponsored formative 
assessments (e.g. DIBELS, QRI, running records, writing prompts, commercial tests), district- 
sponsored summative assessments (e.g. IOWA, Stanford, Terra Nova, portfolios), and formal 
classroom assessments (administered across classrooms) used for grading. ” 

2. Lay out the numbers horizontally from -3 (left) to +3 (right). 

3. Read through all 23 cards to become familiar with the statements. 

4. As you read through the statements for a second time, organize them into three piles: 

• On the right, place the cards that are most characteristic of your current practice regarding 
school data. 

• On the left, place the cards that are least characteristic of your current practice regarding 
school data. 

• In the middle, place the cards that you feel more ambivalent about. 

5. Beginning with the pile on the right (the most characteristic side), place the two cards that are 
most characteristic of your current practice under the +3 marker. 
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6. Now, turning to your left side (least characteristic), place the two cards that are least 
characteristic of your current practice under the -3 marker. 

7. Returning to the pile on the right, choose three cards that are more characteristic of your current 
practice with regard to school data, but are not as much as the two you already selected and 
placed under marker +3, and place these cards under marker +2, in any order. 

8. Do the same with the pile on the left, following this pattern as you work your way to the center 
pile. 

9. If you would have trouble giving examples to the items on the card, they may not rank very 
high. 

10. You are free to change your mind during the sorting process and switch items around, as long 
as you maintain the requested number of items under each marker 

• You should have 2 cards under markers +3 and -3. 

• You should have 3 cards under markers +2 and -2. 

• You should have 4 cards under markers +1 and -1. 

• You should have 5 cards under marker 0. 

11. Your sorted cards should match the diagram below. After sorting the cards, please record the 
number on the cards onto the diagram below in the order in which you placed them. KEEP 
YOUR CARDS OUT—you will need them to answer the follow-up questions. 


-3 

-2 

-1 

0 

+ 1 

+2 

+3 


Post Q Sort Interview Questions: 

1) Please list one card in the +3 column and your reasons for placing it there. 

Card #:_ 

2) Please list one card in the -3 column and your reasons for placing it there. 

Card #: 
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3) Were there specific statements that you had difficulty placing? Choose one and please list the 
number of the statement and describe your dilemma. 

Card #: 


4) What has had the greatest impact on how you sorted your cards the way you did? (Examples—past 
experience, courses, current knowledge, etc.). Please explain your answers. 


5) In order, what are the three most important pieces of data you use as a 
teacher/principal? How and Why? 
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